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3.  Use  of  OTj&E  procedures  was  minimally  effective  for  performance  measurement  but  served  well  as  a means 
for  acquiring  aircrew  subjective  impressions  on  the  system’s  overall  training/evaluation  potential. 
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EVALUATION  OF  THE  AUTOMATED  ADAPTIVE  FLIGHT 
TRAINING  SYSTEM’S  AIR-TO-AIR  INTERCEPT 
PERFORMANCE  MEASUREMENT 


I.  INTRODUCTION 

Background 

The  Automated  Adaptive  Flight  Training 
System  (AAFTS),  developed  by  Logicon,  Inc.,  and 
presently  installed  on  the  F-4E  #15  Weapon 
System  Training  Set  (WSTS)  at  Luke  AFB, 
Arizona,  was  designed  to  provide  an  automated 
capability  to  train  crews  in  ground  controlled 
approaches  (GCAs),  tactical  air  navigation 
approaches  (TACANs),  air-to-air  intercepts  (AAI), 
and  ground  attack  radar  (GAR)  missions. 

An  evaluation  of  the  GCA  portion  of  the 
AAFTS  was  conducted  from  May  to  November 
1975  through  the  joint  efforts  of  Tactical  Air 
Command  (TAC)  and  AFHRL/FT  at  the  request 
of  TAC/DOXS.  Brown,  Waag,  & Eddowes  (1975) 
concluded  that:  (a)  there  were  no  differences  in 
performance  on  the  training  trials  between  the 
AAFTS  groups  and  Standard  groups;  (b)  the  group 
trained  by  the  machine  controller  performed  as 
well  as  the  group  trained  by  human  operators  on 
the  basis  of  criterion  sorties;  and  (c)  the  AAFTS 
training  did  not  appear  to  result  in  any  adverse 
student  GCA  responses  as  measured  during  the 
study. 

An  operational  test  and  evaluation  (OT&E)  was 
scheduled  to  evaluate  other  capabilities  of  the 
AAFTS  which  were  delivered  subsequent  to  the 
initial  GCA  evaluation.  These  included:  (a)  a re- 
evaluation  of  the  GCA  portion  of  the  device;  (b) 
AAI;  (c)  GAR;  (d)  Speech  Understanding  System 
(SUS);  and  (e)  Remote  Terminal  Effectiveness. 
Results  of  the  evaluation  are  available  from  USAF 
TAWC  (TAC  Project  74A  119U),  Eglin  AFB, 
Florida.  The  data  gathered  during  the  course  of 
the  OT&E  provided  the  data  source  for  this  effort. 

Performance  Measurement 

Data  gathered  during  the  OT&E  were  designed 
to  answer  specific  objectives  regarding  device  ef- 
fectiveness for  aircrew  training.  However,  it  was 
expected  these  data  also  would  be  sufficient  for 
use  as  a base  to  evaluate  the  performance  measure- 
ment schemata  developed  to  score  crew 
performances  in  the  AAI  mode. 

In  recent  years,  automated  performance  meas- 
urement (APM)  of  aircrew  skill  has  become  in- 
creasingly important,  due  to  the  emphasis  placed 


upon  simulation  training  and  the  advent  of  large 
capacity  digital  computing  systems  capable  of 
handling  performance  measurement  requirements. 
The  digital  computer  has  the  capability  to  measure 
many  variables  simultaneously  at  a very  rapid  rate 
with  extreme  precision.  Thus,  APM  can  provide  a 
large  performance  data  base  in  a relatively  short 
period  of  time. 

One  of  the  most  difficult  skill  areas  in  which 
APM  has  been  attempted  is  the  air-to-air  combat 
environment.  Selection  and  implementation  of 
variables  that  are  fluid  enough  to  be  adapted  to 
the  ever-changing  environment  of  air-to-air 
combat,  yet  sensitive  enough  to  reliably  discri- 
minate among  various  levels  of  pilot  proficiency, 
have  been  difficult  to  identify.  Therefore,  several 
strategies  for  measurement  of  air  combat  maneu- 
vering (ACM)  have  been  adopted  by  the  research 
community.  While  most  studies  involved  in  this 
problem  area  use  the  ACM  arena  as  a data  source, 
many  of  the  variables  used  in  ACM  are  applicable 
to  the  AAI  environment. 

One  strategy  has  attempted  to  classify  the 
engagement  into  a series  of  individual  maneuvers 
and  to  evaluate  each  of  these  maneuvers  separately 
and  then  summate  the  findings  into  a composite 
score.  A second  strategy  has  attempted  to  select 
several  variables  that  have  intuitive  logical  appeal, 
integrate  them  into  some  composite  scoring 
system,  and  analyze  the  results  (TAC  Project 
74T-912F).  A third  strategy  has  been  to  select  a 
relatively  large  number  of  potentially  relevant 
variables  for  measurement,  fly  engagements,  and 
then  statistically  eliminate  those  variables  not  con- 
tributing to  the  discriminability  of  pilot  perform- 
ance for  a final  reduced  variable  set.  A fourth 
strategy  has  been  to  use  the  computer  as  an  infor- 
mation system  by  measuring  a set  of  preselected 
variables  and  reporting  the  results  without  any 
attempt  to  provide  a composite  score.  A fifth 
strategy  has  been  use  of  energy  management  as  the 
key  variable  in  measurement  of  ACM  success 
(Deberg,  1977).  Evaluation  of  each  of  these  alter- 
natives has  produced  less  than  adequate  results. 
Typically,  each  APM  system  has  strong  and  weak 
points  so  that  no  single  system  has  been  able  to 
completely  replace  a careful,  thorough,  good 
subjective  evaluation  by  an  expert  pilot. 

A final  strategy  for  performance  measurement 
has  been  the  use  of  criterion-referenced  systems. 
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These  systems  usually  have  several  preselected  vari- 
ables to  be  measured  with  err  or  tolerances 
measured  with  bandwidths  assigned  to  each  vari- 
able and  differing  points  awarded  to  aircrews  as  a 
function  of  their  ability  to  maintain  prescribed 
limits  on  the  variables  in  question.  In  the  AAI 
environment,  variables  often  measure  the  ge- 
ometry of  the  attack,  missile  launch  parameters 
and  other  variables  deemed  important  to  an  inter- 
cept (Shepherd,  1975).  The  AAFTS  device  used  a 
scoring  system  of  this  nature.  A set  of  28  de- 
pendent variables  was  determined  by  the  4444 
Operational  Training  Development  Team  (OTDT) 
and  implemented  into  the  computer  by  Logicon. 
These  28  variables  included  measures  of:  How 
closely  the  crew  stayed  to  assigned  airspeeds,  alti- 
tudes and  headings;  how  accurately  the  crew 
launched  missiles;  whether  or  not  correct  missile 
launching  control  procedures  were  followed/ 
accomplished;  and  other  variables  specific  to  parti- 
cular types  of  AAIs.  All  variables  had  acceptable 
bandwidths  of  performance  that  the  crew  had  to 
maintain  in  order  to  accumulate  scoring  points  in 
the  scoring  system. 


II.  METHOD 

Subjects 

Seven  operationally  qualified  crews  and  five 
student  crews  participated  in  the  OT&E  of  the 
AAFTS  system.  The  seven  operational  crews  came 
from  the  following  locations:  two  crews  from 
Luke  AFB  and  one  each  from  Eglin  AFB, 
Holloman  AFB,  Seymour-Johnson  AFB,  Moody 
AFB,  and  Hill  AFB.  All  operational  crews  had 
completed  the  F-4  student  course  and  had  been  on 
operational  duty  an  average  of  12  months.  The 
five  student  crews  were  members  of  the  550th 
Tactical  Fighter  Training  Squadron  (TFTS) 
enrolled  in  Course  4000B  at  Luke  AFB. 

Apparatus 

The  AAFTS  is  a parasitic  computer  based  in- 
structional device  that  has  been  interfaced  with 
F4E  WSTS#15.  The  system  is  capable  of  auto- 
matically scoring  and  advancing  aircrews  through  a 
25-step  GCA  program  (complete  with  computer- 
driven  GCA  instructions),  a 12-step  GAR  program, 
and  a 116-step  AAI  program  (complete  with 
computer-generated  Ground  Controlled  Intercept 
(GCI)  instructions  and  computer-driven  speech 
understanding  capabilities).  The  system  has  two 
cathode  ray  tubes  (CRTs)  with  interactive  key- 
board consoles  to  observe  performance  con- 


currently or  post  hoc.  Hard  copy  printouts  of  the 
tasks  are  available. 

The  AAI  program  is  broken  into  nine  modes, 
each  with  a various  number  of  steps  (AAI  engage- 
ments) within  it.  The  nine  modes  are:  (a)  single 
turn  attacks  (28  steps);  (b)  increasing/decreasing 
aspect  attacks  (12  steps);  (c)  stem  conversion 
attacks  (20  steps);  (d)  stem  conversion  ID  passes 
(10  steps);  (e)  snap-ups  (8  steps);  (0  single  turn 
attacks  with  stem  conversion  reattacks  (6  steps); 
(g)  no  lock-on  attacks  (10  steps);  (h)  no  GCI 
attacks  (10  steps);  and  (i)  tactical  intercepts  (12 
steps).  Within  each  mode,  the  steps  were  designed 
to  present  progressively  more  difficult  intercepts 
as  a function  of  the  values  of  certain  variables. 
These  variables  include  range  of  contact  of  the 
hostile  target,  rate  of  closure  of  the  target,  target 
maneuverability,  altitude  separation,  and  lock-on 
range.  While  the  AAFTS  has  a 28-variable  standard 
format,  only  those  variables  relevant  to  the  parti- 
cular engagement  underway  are  scored  by  the 
AAFTS  APM  system.  Thus,  within  each  mode  of 
attack,  the  same  variables  are  measured  for  all  the 
steps.  However,  across  modes,  different  variables 
are  measured  in  order  to  best  depict  characteristic 
crew  performance  in  the  mode  under  considera- 
tion. Appendix  A lists  all  performance  variables, 
the  scoring  schemata,  and  when  each  variable  is 
measured. 

The  AAFTS,  in  essence,  becomes  the  in- 
structor, since  it  assigns  interceptor  altitude,  speed 
and  heading  and  then  vectors  the  interceptor  to  a 
target  (usually  flying  straight  and  level  at  a parti- 
cular altitude  and  heading).  Upon  completion  of 
each  AAI  engagement,  all  APM  variables  are  avail- 
able for  inspection  via  hard  copy  printout.  The 
total  cumulative  score  is  displayed  on  the  CRT 
immediately  upon  completion  of  the  engagement. 

Procedure 

Training  received  in  the  AAFTS  was  severely 
constrained  by  TAC  training  requirements  in  the 
case  of  student  crews  and  by  OT&E  limitations  in 
the  case  of  the  operational  crews.  The  operational 
crews  were  sent  TDY  to  Luke  AFB,  one  crew  per 
week  for  seven  weeks  to  fly  the  F4EWSTS#15/ 
AAFTS  for  an  average  of  two  hours  a day.  All  five 
student  crews  were  enrolled  in  the  F-4000B  course 
during  which  they  accomplished  25  simulator 
missions.  Simulator  missions  8 through  14 
inclusive  were  devoted  to  AAI  engagements  as 
required  by  the  4000B  course  syllabus.  During 
these  seven  missions,  the  instructors  used  the 
AAFTS  device  for  training  the  student  crews. 
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Computer  advancement  logic  was  not  used,  since 
the  instructors  chose  which  engagements  they 
desired  from  the  AAFTS  repertoire  of  Mode/Step 
possibilities.  Thus,  the  data  gathered  represents  an 
accumulation  of  the  available  number  of  AAls 
completed  by  each  of  the  twelve  crews  used  for 
the  OT&E. 

Experimental  Design  and  Analysis 

To  study  potential  performance  differences 
between  operational  crews  and  student  crews,  the 
data  collection  methods  had  to  be  considered  care- 
fully. The  crews  flew  the  simulator  under  different 
conditions,  since  the  operational  crews  were 
adaptively  scheduled  by  the  AAFTS  as  often  as 
possible,  while  the  student  crews  flew  what  was 
dictated  by  instructors.  Thus,  while  data  were 
accumulated  in  each  mode,  the  specific  steps 
presented  within  each  mode  were  flown  under 
somewhat  different  conditions.  Also,  since  the 
tasks  measured  by  each  mode  were  slightly  dif- 
ferent, each  mode  had  to  be  evaluated  separately. 

T-statistics  were  used  to  test  differences 
between  the  total  scores  (a  summation  of  all  the 
variables  measured  for  each  engagement  com- 
pleted) achieved  by  the  student  crews  and  opera- 
tional crews  for  each  mode.  They  were  also  used 
to  test  differences  between  operational  and 
student  crew  performance  on  a variable-by-variable 
basis  across  modes,  wherever  the  particular 
variable  was  measured. 

The  AAFTS  device  had  specific  point  values 
assigned  to  performance  error  tolerance  limits  for 
each  variable,  as  evidenced  in  Appendix  A.  An 
equal  interval  scoring  technique  was  also  employed 
to  more  accurately  satisfy  assumptions  on  which 
the  statistic  is  based  (i.e.,  a variable  with  scored 
values  of  10,  8,  6,  3,  0 was  arbitrarily  rescaled  to 
4,  3,  2,  1 , 0),  and  then  between-groups  differences 
were  examined  by  mode  and  by  variable. 

m.  RESULTS 

Table  1 displays  the  mean,  standard  deviation, 
and  t-statistic  for  each  mode  of  attack  summed 
across  the  operational  and  student  crews,  using  the 
percent  of  possible  total  score  variable  (i.e.,  a com- 
posite score  of  all  variables  measured  for  each 
AAI) 

Modes  1 (single-turn  attacks)  and  3 (stern- 
conversion  attacks)  produced  significant  perform- 
ance differences  between  student  and  operational 
crews.  The  better  performance,  p < .01  and  p < 
.05,  respectively,  was  achieved  by  the  operational 
group  in  both  modes. 


Descriptions  of  student  crew  and  operational 
crew  performance  for  each  variable  are  in  Ap- 
pendices B and  C.  These  appendices  show  the 
frequencies  of  scoring  achieved  by  variable  for 
student  and  operational  crews,  respectively,  for  all 
the  dependent  variables  across  all  modes  of 
intercepts. 

Table  2 shows  that  performance  differences 
between  the  operational  and  student  crews  on  the 
eighth,  (closure  speed),  22nd  (rollout  aspect  angle 
differential),  and  23rd  (stern  rollout,  overtake 
speeds)  variables,  using  the  AAFTS  scoring  con- 
figuration, were  significant  (p  < .05).  The  opera- 
tional crews  performed  better  on  the  eighth 
variable,  while  the  student  crews  showed  the 
better  performance  on  the  22nd  and  23rd  vari- 
ables. When  the  equal  interval  scoring  was  com- 
puted, few  changes  were  observed  in  significant 
findings  on  a variable-by-variable  basis,  as 
evidenced  in  Table  2. 

The  eighth,  22nd,  and  23rd  variables  again  were 
significant  (p  < .05),  using  the  equal  interval 
scoring  system.  The  fifth  variable  (aim  dot  align- 
ment error),  which  showed  a performance  differ- 
ence of  p < .1 0 using  the  AAFTS  scoring,  changed 
to  p < .05,  using  equal  interval  scoring.  Individual 
crew  means  used  for  comparisons  of  the  AAFTS 
and  equal-interval  scoring  are  available  in 
Appendices  D and  E,  respectively. 

IV.  DISCUSSION 

Modes  1 and  3 which  showed  better  perform- 
ance by  the  operational  crews  possibly  can  be  ex- 
plained as  a function  of  their  greater  familiarity 
with  air-to-air  intercepts.  The  operational  crews, 
by  virtue  of  already  having  completed  the  student 
course,  were  more  attuned  to  the  air-to-air  en- 
vironment than  were  the  student  crews.  The 
student  crews  usually  flew  several  single-turn  con- 
version intercepts  (Mode  1),  and  if  successful  on 
those,  in  the  judgment  of  the  instructor,  flew  addi- 
tional stem  conversion  intercepts  (Mode  3),  as 
time  permitted,  in  accordance  with  syllabus  re- 
quirements. These  were  the  first  intercepts  flown 
by  the  students  in  a simulator  environment.  Con- 
versely, while  the  operational  crews  flew  single- 
turn  conversions  first  in  the  simulator  also,  it  was 
not  their  first  exposure  to  intercepts.  Thus,  the 
operational  crews  flew  comparatively  superior 
intercepts  in  Mode  1 . The  operational  crews  also 
performed  more  successfully  in  Mode  3,  as  that 
particular  mode  of  attack  demanded  extremely 
rapid  computations  from  the  weapon’s  system 
officer  (WSO),  in  order  to  vector  the  aircraft 
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Table  1.  Means,  Standard  Deviation  and  t-$tatistic  by  Mode  Across  Crews 
for  the  %-Possible  Total  Score  Variable 


Mod* 

*<Op) 

o(Opl 

X(Stud) 

o(Stud) 

t-Value 

P 

i 

89.9 

13.3 

73.2 

29.4 

3.22 

.009** 

2 

79.8 

22.7 

84.4 

12.0 

-1.07 

.31 

3 

813 

16.4 

72.3 

21.7 

2.41 

.035* 

4 

55.7 

23.6 

67.4 

10.5 

-1.92 

.08 

5 

67.6 

33.6 

A 

A 

6 

64.8 

24.9 

73.6 

21.8 

-1.82 

.09 

7 

68.2 

14.6 

63.1 

263 

.69 

.51 

8 

68.0 

27.9 

A 

A 

9 

63.2 

32.6 

52.4 

34.8 

1.10 

.29 

•p  < .05. 

**p<  .01. 

Op  = 

Operational  crews. 

Stud  = 

Student  crews. 

A = 

No  data. 

Table  2.  Comparison  of  Student  and  Operational  Crew  Performances, 

Using  AAFTS  and  Equal-Interval  Scoring  by 

Variable  Across  Modes  via  t-Statistics 

AAFTS  Scoring 

Equal  Interval  Scoring 

Variable 

X(Stud) 

S(Op) 

t 

X(Stud) 

X(Op) 

t 

Var#  1 

7.17 

5.92 

1.69 

2.78 

2.28 

1.72 

Var#  2 

9.13 

8.49 

1.16 

2.65 

2.39 

1.28 

Var#  3 

9.14 

9.68 

-2.00 

2.73 

2.91 

-1.78 

Var  # 4 

1.60 

3.27 

—0.85 

0.21 

0.45 

-0.86 

Var#  5 

5.82 

7.46 

-2.02 

2.23 

2.90 

-2.04* 

Var#  6 

8.10 

8.59 

-0.68 

0.81 

0.86 

-0.68 

Var#  7 

13.88 

15.29 

-0.95 

1.99 

2.21 

0.86 

Var#  8 

3.59 

4.40 

-2.55* 

1.43 

1.74 

-2.50* 

Var#  9 

10.48 

11.66 

-1.39 

2.10 

2.33 

138 

Var  #10 

3.85 

4.20 

-1.16 

0.77 

0.84 

-1.18 

Var  #11 

32.30 

34.74 

-0.89 

1.01 

1.74 

-0.97 

Var  #12 

6.29 

7.31 

-0.98 

0.63 

0.73 

-0.98 

Var  #13 

11.70 

8.57 

1.74 

1.64 

1.17 

1.79 

Var  #14 

2.69 

1.65 

1.10 

0.84 

0.53 

1.07 

Var  #15 

7.60 

8.39 

-0.83 

0.76 

0.84 

-0.83 

Var  #16 

11.79 

10.68 

0.97 

1.91 

1.87 

0.29 

Var  #17 

8.64 

8.32 

0.55 

0.86 

0.83 

0.56 

Var#18 

23.03 

27.81 

-133 

1.15 

1.39 

-133 

Var  #19 

5.29 

3.81 

1.06 

0.53 

0.38 

1.05 

Var  #20 

1.15 

2.37 

-1.48 

0.98 

2.37 

-1.63 

Var  #21 

4.50 

1.64 

2.01 

1.50 

0.82 

1.17 

Var  #22 

4.93 

0.76 

4.97* 

2.47 

038 

4.97* 

Var  #23 

6.56 

1.60 

3.%* 

3.72 

1.35 

2.81* 

Var  #24 

2.33 

3.01 

-0.36 

0.23 

0.30 

-0.36 

Var  #25 

8.95 

8.65 

0.65 

0.90 

0.87 

0.66 

Var  #26 

28.21 

27.28 

0.88 

0.94 

0.91 

0.89 

Var  #27 

32.54 

32.80 

-0.10 

0.81 

0.82 

-0.10 

Var  #28 

5.98 

3.54 

1.45 

1.20 

0.71 

1.45 

«p  < .05. 

Stud  ■ Student  crews. 

Op  = Operational  crews. 
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correctly.  Again,  familiarity  with  the  intercepts 
probably  allowed  the  operational  crews  to  score 
more  effectively. 

The  remaining  modes  of  attack  showed  little 
differences  between  operational  crews  and  student 
crews.  This  finding  can  be  attributed  to  two  pos- 
sibilities. First,  according  to  the  informal  sub- 
jective reports  of  operational  crews,  they  did  not 
practice  specific  types  of  intercepts  frequently. 
Rather,  they  were  allowed  the  freedom  of  com- 
pleting the  intercept  in  the  manner  they  deter- 
mined to  be  most  acceptable.  Thus,  their 
performance  on  the  specific  attacks  desired  by  the 
AAFTS  was  not  that  much  superior  to  the  student 
crews,  who,  althou$i  they  were  experiencing  the 
intercepts  for  the  first  time,  learned  rapidly  and 
then  achieved  comparable  scores.  Secondly,  the 
measurement  scheme  employed  by  the  AAFTS 
device  may  not  have  measured  certain  subtle  but 
important  differences  between  the  two  sets  of 
crews  that  could  have  changed  the  performance 
ratings  achieved  by  the  crews,  or  the  scoring  may 
not  have  properly  weighted  variables  that  would 
have  shown  differences  between  the  crews. 

Examination  of  performance  by  both  sets  of 
crews  on  a variable-by-variable  basis  revealed  some 
differences  in  performance.  The  fifth  variable,  aim 
dot  alignment  error,  showed  superior  performance 
by  the  operational  crews  under  both  systems  of 
scoring.  The  exact  probabilities,  p < .068  for  the 
AAFTS  scoring  and  p < .049  using  equal  interval 
scoring,  were  similar.  As  evidenced,  the  opera- 
tional crews  placed  the  target  display  more  pre- 
cisely in  the  center  of  the  allowable  steering  error 
(ASE)  circle,  which  was  critical  in  achieving  a 
missile  kill.  As  accuracy  in  placement  of  the  target 
display  on  the  center  of  the  radar  scope  of  the 
interceptor  increased,  the  probability  of  achieving 
the  kill  also  increased. 

Operational  crews  performed  better  on  the 
eighth  variable,  closure  speed  error,  than  did 
student  crews  with  virtually  no  difference  between 
the  probability  levels  of  the  tests,  as  a function  of 
type  of  scoring,  p < .027  and  p < .030,  AAFTS 
scoring  and  equal  interval  scoring,  respectively.  In 
many  intercepts  the  student  crews  were  not 
attuned  to  the  voice  generation  unit  of  the  AAFTS 
and  missed  some  of  the  vectoring  instructions. 
Alternately,  the  student  crews  also  listened  too 
intently  to  the  breakaway  heading  assignment, 
which  was  the  last  instruction  given  in  a long  series 
where  one  of  the  first  instructions  was  the  assigned 
vectoring  instructions.  This  sequencing  caused 
some  indecision  on  what  the  appropriate  heading 


should  have  been  on  the  part  of  student  crews. 
These  two  problems  often  forced  the  student 
crews  to  increase  speed  in  order  to  reduce  the 
probability  of  an  extended  tail  chase  to  complete 
the  intercept , which  resulted  in  lower  scores  as  the 
assigned  parameters  for  closure  speed  were  vio- 
lated. More  thorough  inbriefing  of  the  student 
crews  on  the  voice  qualities  and  sequencing  of  the 
computer-driven  instructions  might  have  alleviated 
their  problems. 

The  student  crews  performed  better  than  the 
operational  crews  on  the  rollout  aspect  angle 
variable  (#22)  and  the  stem  rollout  overtake  speed 
variable  (#23).  The  difference  between  the 
AAFTS  and  equal  interval  scoring  was  nonexistent 
for  the  rollout  aspect  angle  variable,  but  rather 
noticeable  or.  the  rollout  overtake  speed  variable 
with  p < .022  and  p < .004  for  the  AAFTS  and 
equal  interval  scoring,  respectively.  This  change  in 
the  probability  levels  of  the  statistical  tests  may  be 
due  to  the  fact  that  AAFTS  scoring  awarded  in- 
creasingly higher  point  values  for  being  within  the 
specified  tolerances  as  the  target  aircraft  was 
approached.  Thus,  the  AAFTS  scoring  was  posi- 
tively accelerated  as  the  target  aircraft  was 
approached,  whereas  the  equal  interval  scoring  was 
linearly  accelerated  as  the  target  was  approached. 

The  rationale  for  student  superiority  on  these 
two  variables  was  as  follows.  As  previously  men- 
tioned, student  crews  often  “got  behind”  on 
stern-conversion  intercepts  (which  was  the  only 
attack  element  on  which  variables  22  and  23  were 
measured;  see  Appendix  A).  The  student  crews 
frequently  did  end  up  in  tail  chases  which  put 
them  directly  behind  the  target  aircraft  and  kept 
them  well  within  the  tolerances  desired  by  the 
rollout,  aspect  angle  variable.  Improved  perform- 
ance by  the  student  crews  on  the  rollout  overtake 
speed  variable  occurred  in  a similar  manner.  Once 
behind  the  target  aircraft  in  a tail  chase,  the 
student  crews  had  more  time  to  more  accurately 
monitor  their  overtake  rates  and,  accordingly,  ad- 
justed to  the  desired  rates  of  closure.  Since  the 
AAFTS  had  no  scoring  for  time-to-kill  or  the  over- 
all geometry  of  the  attack,  any  initial  errors  made 
by  student  crews  were  offset  by  the  increased  time 
expended  in  a tail  chase,  during  which  all  re- 
maining variables  to  be  scored  were  performed 
well  by  the  student  crews.  However,  it  must  be 
remembered  that  operational  crews  usually  per- 
formed more  geometrically  appropriate  and  more 
rapid  kills.  Their  overall  performance  was  usually 
superior  to  student  crew  performance  which  was 
probably  a function  of  their  superior  ability  to 
conceptualize  the  requirements  of  (he  entire 


9 


intercept  rather  than  reacting  to  specific  problems 
throughout  the  intercept. 

In  fact,  there  were  relatively  few  differences 
shown  as  a function  of  scoring.  This  was  due  in 
part  to  the  large  percentage  of  dichotomous  (two- 
level)  variables  present  in  the  28-variable  format 
used  by  the  AAFTS.  As  the  equal-interval  scoring 
was  a strict  linear  transformation  of  the  scored 
points,  the  t-values  associated  with  all  dichot- 
omous variables  were  identical  (within  rounding 
error  to  the  AAFTS  scoring  t-values).  Of  the 
remaining  multiple-level  variables,  differences  were 
also  slight,  since  the  change  from  the  AAFTS 
scoring  to  equal-interval  scoring  was  simply  not 
that  dramatic. 

The  AAFTS  system  performed  adequately  as  a 
computer  information  system.  On  a variable- 
by-variable  basis,  the  AAFTS  wa..  at'e  to  provide 
useful  information  regarding  those  parameters 
being  measured;  however,  the  combination  of 
those  individual  variable  scores  into  a total  score 
that  could  discriminate  between  varying  levels  of 
crew  sophistication  was  inadequate.  In  order  to 
evaluate  the  28-variable  format  empirically  so  that 
meaningful  coefficient  weights  could  be  assigned 
to  the  variables,  each  crew  should  have  completed 
a selected  number  of  mode/step  combinations 
many  times  to  provide  a good  data  base.  Sufficient 
time  to  complete  that  requirement  was  simply  not 
available  in  the  OT&E  environment. 

By  virtue  of  the  points  assigned  to  the  variables, 
a crude  attempt  at  coefficient  weighting  was  al- 
ready available  within  the  AAFTS  system.  This 
system,  however,  decreased  the  probability  of 
observing  distinctions  between  crew  skill  levels  due 
to  its  specific  coefficient  weighting.  For  example, 
variables  26  and  27  accounted  for  a composite  of 
70  points.  Virtually  all  the  crews  achieved  a kill 
(40  points— variable  27),  and,  similarly,  few  of  the 
pilots  crashed  their  simulated  aircraft  during  the 
course  of  an  intercept  (30  points-variable  26) 
Further,  most  pilots  correctly  configured  their 
switches  a very  high  percentage  of  the  time  (80 
points-variables  11,  18).  Thus,  a score  of  150 
points  with  little  variance  between  the  crews  was 
achieved.  The  remaining,  measured  variables 
usually  had  no  more  than  a maximum  of  ten 
points  per  variable,  again  with  relatively  small 
variances  of  performance  across  both  operational 
and  student  crews.  The  net  result  was  a lack  of 
discrimination  between  the  student  and  opera- 
tional crews  (i.e.,  good  performance  measurement) 
due  to  preselection  of  variable  weights  and  little 
empirical  validation  of  those  assigned  variable 
weights. 


V.  CONCLUSIONS/RECOMMENDATIONS 

1 . The  scoring  system  of  the  AAFTS  device 
faithfully  provided  excellent  information  about 
intercepts  via  hard  copy  records  and  CRT  replays 
of  the  AAls.  This  feature  alone  was  obviously 
superior  to  crew  or  instructor  memory,  parti- 
cularly in  debriefing  situations.  This  system  of 
scoring  should  be  considered  to  be  a first  attempt 
with  an  equivalent  level  of  sophistication.  The 
information  provided  was  useful,  but  the  integra- 
tion of  the  variables,  complete  with  appropriate 
empirical  weighting,  into  a single,  total  score 
variable  that  would  discriminate  between  skill 
levels  of  a crew  is  inadequate.  Future  efforts 
would  require  more  extensive  testing.  Use  of  two- 
level  variables  and  discrete  multiple-level  variables 
provides  relatively  small  variances,  as  opposed  to 
variables  that  are  allowed  to  change  on  a con- 
tinuous scale.  A system  with  more  discriminative 
power  would  probably  be  forced  to  utilize  several 
continuous  variables,  in  order  to  provide  more 
detailed  information  on  crew  performance. 
Additionally,  variables  that  measure  time-to-kili 
and  the  overall  geometry  of  the  attack  should  be 
added. 

2.  The  OT&E  test  environment  provided  a 
quick  look  into  specific  strengths  and  weakness  of 
the  AAFTS  as  a training  system.  These  data  were 
usually  subjective  and  gathered  via  student/ 
instructor  commentary  to  the  experimenter.  Use 
of  OT&E  procedures  to  conduct  a thorough  per- 
formance measurement  evaluation  was  extremely 
difficult.  By  attempting  to  maintain  training  pro- 
grams without  modification,  the  data  requirement 
for  a thorough  performance  measurement 
evaluation  was  compromised. 

3 . The  potential  for  this  system  to  be  used  as  a 
standardized  evaluative/training  tool  for  simula- 
tion is  excellent.  In  the  future,  the  system  could 
be  configured  to  completely  automate  either 
training  or  testing  scenarios.  Test  and  training 
conditions  would  be  guaranteed  to  be  identical  as 
a function  of  the  computer  and  could  be  modified 
as  necessary.  With  some  further  analysis  and 
change  of  the  variables  under  consideration,  the 
objective  scoring  could  be  improved  sufficiently  to 
provide  satisfactory  objective  measures  of  crew 
performance.  This  scoring  could  be  made  to 
conform  to  desired  learning  outcomes  of  various 
syllabi  for  different  aircraft. 
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APPENDIX  A.  AAI  PERFORMANCE  MEASUREMENT  VARIABLES 
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APPENDIX  B.  FREQUENCY  DISTRIBUTION  OF  STUDENT  CREW 

SCORING  BY  VARIABLE  ACROSS  MODES 

Pouibto 

1st 

2nd 

3rd 

4th 

9th 

Variable 

Scorn 

Craw 

Craw 

Craw 

Craw 

Craw 

Total 

i 

10 

9 

41 

22 

17 

20 

109 

8 

5 

12 

14 

14 

7 

52 

6 

4 

1 

4 

7 

6 

22 

3 

2 

0 

0 

1 

1 

4 

0 

11 

9 

11 

4 

3 

38 

Total 

31 

63 

51 

43 

37 

225 

2 

10 

20 

61 

52 

34 

39 

206 

8 

5 

1 

1 

7 

2 

16 

6 

3 

4 

0 

5 

2 

14 

0 

5 

2 

1 

2 

0 

10 

Total 

33 

68 

54 

48 

43 

246 

3 

10 

26 

58 

50 

35 

34 

203 

8 

0 

0 

0 

1 

0 

1 

• 

6 

0 

1 

0 

1 

0 

2 

0 

5 

4 

1 

6 

1 

17 

■ 

Total 

31 

63 

51 

43 

35 

223 

4 

20 

0 

3 

0 

0 

0 

3 

15 

0 

1 

1 

0 

0 

2 

10 

0 

2 

1 

0 

0 

3 

0 

5 

10 

10 

0 

11 

36 

Total 

5 

16 

12 

0 

11 

44 

5 

10 

6 

13 

19 

4 

8 

50 

8 

12 

10 

12 

6 

10 

50 

6 

5 

5 

5 

4 

7 

26 

3 

2 

0 

1 

2 

3 

8 

: 

0 

10 

10 

10 

16 

9 

55 

Total 

35 

38 

47 

32  ‘ 

37 

189 

6 

10 

25 

29 

39 

21 

28 

142 

0 

0 

7 

8 

11 

9 

35 

Total 

25 

36 

47 

32 

37 

177 

7 

20 

13 

20 

24 

3 

17 

77 

15 

12 

3 

12 

17 

12 

56 

10 

0 

0 

3 

0 

1 

4 

0 

3 

7 

8 

12 

7 

37 

Total 

28 

30 

47 

32 

37 

174 

8 

5 

17 

28 

34 

20 

30 

129 

3 

3 

0 

2 

0 

0 

5 

0 

8 

10 

11 

12 

7 

48 

Total 

28 

38 

47 

32 

37 

182 

9 

20 

1 

1 

3 

1 

2 

8 

15 

14 

18 

23 

13 

22 

90 

10 

10 

8 

13 

6 

5 

42 

5 

0 

15 

0 

1 

1 

17 

0 

3 

10 

8 

11 

7 

39 

Total 

28 

52 

47 

32 

37 

196 

19 

1 
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Appendix  B ( Continued ) 


Variable 

Possible 

Scorn 

1st 

Craw 

2nd 

Craw 

3rd 

Craw 

4th 

Craw 

9th 

Craw 

Total 

10 

5 

25 

27 

38 

20 

30 

140 

0 

3 

11 

9 

12 

7 

42 

Total 

28 

38 

47 

32 

37 

182 

11 

40 

24 

28 

38 

21 

30 

141 

30 

1 

1 

1 

0 

0 

3 

0 

0 

9 

8 

11 

7 

35 

Total 

25 

38 

47 

32 

37 

179 

12 

10 

24 

25 

29 

9 

27 

114 

0 

4 

13 

18 

23 

10 

68 

Total 

28 

38 

47 

32 

37 

182 

13 

20 

6 

7 

1 

5 

4 

23 

15 

2 

15 

3 

7 

8 

35 

10 

0 

2 

2 

2 

1 

7 

0 

1 

5 

8 

9 

3 

26 

Total 

9 

29 

14 

23 

16 

91 

14 

10 

0 

1 

1 

5 

7 

14 

6 

1 

7 

0 

3 

4 

15 

3 

0 

4 

3 

2 

0 

9 

0 

8 

17 

9 

13 

5 

52 

Total 

9 

29 

13 

23 

16 

90 

15 

1 

8 

25 

7 

16 

13 

69 

0 

1 

4 

6 

7 

3 

21 

Total 

9 

29 

13 

23 

16 

90 

16 

20 

5 

6 

7 

13 

7 

37 

10 

3 

16 

1 

2 

5 

27 

5 

0 

2 

0 

2 

0 

4 

0 

1 

5 

6 

6 

4 

22 

Total 

9 

29 

13 

23 

16 

90 

17 

10 

21 

37 

25 

26 

22 

131 

0 

1 

11 

2 

7 

3 

24 

Total 

22 

48 

27 

33 

25 

155 

18 

40 

0 

11 

6 

14 

9 

40 

20 

8 

13 

1 

3 

3 

28 

0 

1 

5 

6 

6 

4 

22 

Total 

9 

29 

13 

23 

16 

90 

19 

10 

8 

14 

0 

12 

12 

46 

0 

1 

15 

13 

11 

4 

44 

Total 

9 

29 

13 

23 

16 

90 

20 
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APPENDIX  C.  FREQUENCY  DISTRIBUTION  OF  OPERATIONAL  CREW 
SCORING  BY  VARIABLE  ACROSS  MODES 


PoulMa  Itt  2nd  3rd  4th  Sth  4th  7th 

Vartabln  Scoraa  Craw  Craw  Craw  Craw  Craw  Craw  Craw  Total 

1 10  5 11  5 17  11  9 7 65 

8 6277344  33 

6 2234552  23 

3 1 3 2 1 0 3 1 11 

0 3 6 7 1 16  11  9 S3 

Total  17  24  24  30  35  32  23  185 

2 10  5 26  20  18  27  26  23  145 

8 6201742  22 

6 2229880  31 

0 3 1 2 4 3 1 0 14 


1 


Appendix  C ( Continued ) 


VavlaMa 

Poulbl* 

Scorn 

lat 

Crow 

2nd 

Crow 

3rd 

Crow 

4th 

Crow 

5th 

Crew 

5th 

Crow 

7th 

Crow 

Total 

10 

5 

n 

21 

19 

26 

21 

29 

22 

149 

0 

2 

4 

1 

1 

10 

11 

3 

32 

Total 

13 

25 

20 

27 

31 

40 

25 

181 

11 

40 

11 

23 

20 

26 

22 

29 

22 

153 

30 

0 

0 

0 

0 

0 

0 

0 

0 

0 

2 

2 

0 

1 

9 

11 

2 

27 

Total 

13 

25 

20 

27 

31 

40 

24 

180 

12 

10 

11 

22 

9 

21 

21 

27 

21 

132 

0 

2 

3 

11 

7 

10 

13 

4 

50 

Total 

13 

25 

20 

28 

31 

40 

25 

182 

13 

20 

2 

1 

1 

0 

1 

2 

0 

7 

15 

1 

1 

1 

5 

13 

3 

4 

28 

10 

0 

2 

0 

0 

1 

0 

0 

3 

0 

2 

3 

4 

5 

5 

8 

2 

29 

Total 

5 

7 

6 

10 

20 

13 

6 

67 

14 

10 

0 

0 

0 

0 

3 

2 

0 

5 

6 

2 

1 

0 

0 

2 

2 

1 

8 

3 

2 

1 

0 

1 

2 

0 

1 

7 

0 

1 

5 

6 

9 

13 

10 

4 

48 

Total 

5 

7 

6 

10 

20 

14 

6 

68 

15 

10 

4 

7 

5 

8 

20 

8 

4 

56 

0 

0 

0 

1 

2 

0 

6 

2 

11 

Total 

4 

7 

6 

10 

20 

14 

6 

67 

16 

20 

1 

1 

1 

2 

9 

5 

2 

21 

10 

2 

6 

4 

3 

10 

5 

2 

32 

5 

1 

0 

0 

2 

0 

0 

0 

3 

0 

1 

0 

1 

2 

0 

4 

2 

10 

Total 

5 

7 

6 

9 

19 

14 

6 

66 

17 

10 

5 

5 

11 

20 

36 

21 

16 

114 

0 

0 

2 

4 

4 

5 

8 

1 

24 

Total 

5 

7 

15 

24 

41 

29 

17 

138 

18 

40 

3 

5 

5 

7 

10 

1 

2 

33 

20 

1 

2 

0 

1 

10 

9 

2 

25 

0 

1 

0 

1 

1 

0 

4 

2 

9 

Total 

5 

7 

6 

9 

20 

14 

6 

67 

19 

10 

3 

3 

3 

2 

5 

6 

2 

24 

0 

2 

4 

4 

8 

15 

8 

4 

45 

Total 

5 

7 

7 

10 

20 

14 

6 

69 

20 

6 

0 

0 

0 

1 

0 

0 

0 

1 

5 

0 

0 

0 

0 

1 

0 

0 

1 

4 

0 

0 

1 

0 

2 

0 

1 

4 

3 

1 

0 

0 

0 

0 

1 

0 

2 

2 

0 

1 

2 

0 

0 

0 

0 

3 

1 

0 

0 

0 

0 

0 

0 

0 

0 

23 

Appendix  C ( Continued ) 


Varlabla 

Pottlbl* 

Scorn 

1st 

Craw 

2nd 

Craw 

3rd 

Craw 

4th 

Craw 

5th 

Craw 

6th 

Craw 

7th 

Craw 

Total 

0 

2 

2 

2 

2 

0 

0 

0 

8 

Total 

3 

3 

5 

3 

3 

l 

i 

19 

21 

8 

0 

0 

1 

0 

0 

0 

0 

1 

6 

0 

0 

2 

0 

1 

0 

0 

3 

4 

1 

0 

1 

1 

1 

0 

0 

4 

2 

0 

1 

0 

0 

0 

0 

0 

1 

0 

2 

2 

1 

2 

1 

1 

1 

10 

Total 

3 

3 

5 

3 

3 

1 

1 

19 

22 

6 

0 

0 

0 

0 

0 

0 

0 

0 

4 

C 

0 

0 

0 

0 

0 

1 

1 

2 

0 

0 

0 

2 

0 

0 

0 

2 

0 

3 

3 

5 

1 

3 

1 

0 

16 

Total 

3 

3 

5 

3 

3 

1 

1 

19 

23 

8 

0 

0 

0 

0 

1 

0 

0 

1 

6 

0 

0 

2 

0 

1 

0 

0 

3 

5 

0 

0 

0 

0 

1 

1 

0 

2 

4 

0 

1 

1 

0 

1 

0 

0 

3 

2 

0 

0 

3 

8 

0 

3 

0 

14 

1 

0 

0 

0 

0 

0 

1 

0 

1 

» 

0 

0 

1 

0 

2 

6 

2 

0 

11 

Total 

0 

2 

6 

10 

10 

7 

0 

35 

24 

10 

0 

1 

0 

3 

0 

2 

2 

8 

0 

0 

1 

2 

8 

0 

4 

0 

15 

Total 

0 

2 

2 

11 

0 

6 

2 

23 

25 

10 

14 

28 

27 

30 

48 

20 

24 

191 

0 

6 

5 

0 

4 

3 

4 

4 

26 

ji  Total 

20 

33 

27 

34 

51 

24 

28 

217 

26 

30 

19 

28 

26 

29 

45 

40 

28 

215 

0 

1 

5 

1 

5 

6 

6 

0 

24 

Total 

20 

33 

27 

34 

51 

46 

28 

239 

27 

40 

14 

24 

24 

27 

43 

36 

27 

195 

0 

6 

8 

3 

7 

8 

9 

1 

42 

Total 

20 

32 

27 

34 

51 

45 

28 

237 

28 

10 

0 

0 

0 

4 

1 

3 

2 

10 

5 

0 

0 

0 

0 

1 

2 

0 

3 

0 

0 

0 

0 

2 

0 

1 

3 

6 

Total 

— 

0 

0 

0 

6 

2 

6 

5 

19 

APPENDIX  D.  STUDENT  AND  OPERATIONAL  CREW  PERFORMANCE  BY  VARIABLE  ACROSS 
MODES  COMPARED  BY  T-STATISTIC  USING  AAFTS  SCORING 


Variable 

1ft 

Craw 

2nd 

Craw 

3rd  4th 

Craw  Craw 

5th 

Craw 

6th 

Craw 

7th 

Craw 

1 

Stud  Crew  Xs 

5.16 

8.13 

6.98  7.60 

7.97 

Op  Crew  Xs 

6.65 

6.13 

5.42  8.43 

4.69 

5.03 

5.09 

X(Student)  = 

= 7.17 

X(  Operational)  = 

5.92 

t = 

= 1.69 

p < .1 13  df  = 

10 

2 

Stud  Crew  Xs 

7.82 

9.44 

9.78  8.88 

9.72 

Op  Crew  Xs 

6.88 

9.29 

8.83  7.57 

8.31 

8.72 

9.84 

X (Student) 

= 9.13 

X (Operational)  = 

8.49 

t 

= 1.16 

p < .273  df  = 

10 

3 

Stud  Crew  Xs 

8.39 

9.30 

9.80  8.49 

9.71 

Op  Crew  Xs 

10.00 

9.63 

9.58  9.64 

9.52 

9.36 

10.00 

X (Student) 

= 9.14 

X (Operational)  = 

9.68 

t 

= -2.00 

p < .07 1 df  = 

10 

4 

Stud  Crew  Xs 

0.00 

5.94 

2.08  0.00 

0.00 

Op  Crew  Xs 

7.50 

0.00 

1 .66  0.00 

8.75 

5.00 

0.00 

X (Student) 

= 1.60 

X (Operational)  = 

3.27 

t 

= -0.85 

p < .580  df  = 

10 

5 

Stud  Crew  Xs 

6.86 

6.32 

6.79  3.69 

5.46 

Op  Crew  Xs 

7.23 

8.88 

8.20  8.42 

5.74 

5.33 

8.32 

X (Student) 

= 5.82 

X (Operational)  = 

7.46 

t 

= -2.02 

p < .068  df  = 

10 

6 

Stud  Crew  Xs 

10.00 

8.06 

8.30  6.56 

7.57 

Op  Crew  Xs 

8.4C 

9.20 

10.0  9.63 

6.77 

7.25 

8.80 

X (Student) 

= 8.10 

X (Operational)  = 

8.59 

t 

= -.068 

p < .517  df  = 

10 

7 

Stud  Crew  Xs 

15.71 

14.83 

14.68  9.84 

14.32 

Op  Crew  Xs 

IS. 39 

18.20 

16.00  18.33 

11.61 

11.88 

15.60 

X (Student) 

= 13.88 

X (Operational)  = 

15.29 

t 

= -0.95 

p < .632  df  = 

10 

8 

Stud  Crew  Xs 

3.36 

3.68 

3.74  3.13 

4.05 

Op  Crew  Xs 

4.23 

4.60 

5.00  4.85 

3.10 

4.67 

4.32 

X (Student) 

= 3.59 

X (Operational)  = 

4.40 

*t 

= -2.55 

p < .027  df  = 

10 

9 

Stud  Crew  Xs 

11.79 

8.56 

11.81  8.75 

11.49 

Op  Crew  Xs 

11.54 

12.00 

13.00  11.85 

10.32 

9.72 

13.20 

X (Student) 

= 10.48 

X (Operational)  = 

11.66 

t 

= -1.39 

p < .192  df  = 

10 

10 

Stud  CrewXs 

4.46 

3.55 

4.04  3.13 

4.05 

Op  Crew  Xs 

4.23 

4.20 

4.75  4.82 

3.39 

3.63 

4.40 

X (Student) 

= 3.85 

X (Operational)  = 

4.20 

t 

= -1.16 

p < .270  df  = 

10 

25 


Appendix  D ( Continued ) 


i 


Variable 

1st 

Craw 

2nd 

Craw 

3rd  4th 

Craw  Craw 

5th 

Craw 

Cth 

Craw 

7th 

Craw 

n 

Stud  Crew_Xs 

39.60 

30.26 

32.98  26.25 

32.43 

Op  Crew  Xs 

33.85 

36.80 

40.00  38.52 

28.39 

29.00 

36.67 

X (Student) = 

= 32.30 

X (Operational)  = 

34.74 

t = 

= -0.89 

p < .602  df  = 

10 

12 

Stud  Crew_Xs 

8.57 

6.58 

6.17  2.81 

7.30 

Op  Crew  Xs 

8.46 

8.80 

4.50  7.50 

6.77 

6.75 

8.40 

X (Student) = 

= 6.29 

X (Operational)  = 

7.31 

t = 

= -0.98 

p < .648  df  = 

10 

13 

Stud  CrewXs 

16.67 

13.28 

6.07  9.35 

13.13 

Op  Crew  Xs 

11.00 

7.86 

5.83  7.50 

11.25 

6.54 

10.00 

X (Student)  = 

= 11.70 

X (Operational)  = 

8.57 

t = 

= 1.74 

p < .192  df  = 

10 

14 

Stud  Crew  Xs 

.66 

2.21 

1.46  3.22 

5.88 

Op  Crew  Xs 

3.60 

1.29 

0.00  033 

2.40 

2.43 

1.50 

X (Student)  = 

= 2.69 

X (Operational)  = 

1.65 

t = 

= 1.10 

p < .297  df  = 

10 

15 

Stud  Crew  Xs 

8.89 

8.62 

5.38  6.96 

8.13 

Op  Crew  Xs 

10.00 

10.00 

8.33  8.00 

10.00 

5.71 

6.67 

X (Student)  = 

= 7.60 

X (Operational)  = 

8.39 

t = 

= -0.83 

p < .569  df  = 

10 

16 

Stud  Crew_Xs 

14.44 

10.00 

10.00  12.61 

11.88 

Op  Crew  Xs 

9.00 

11.43 

10.00  8.88 

14.73 

10.71 

10.00 

X (Student)  = 

= 11.79 

X (Operational)  = 

10.68 

t = 

= 0.97 

p < .643  df  = 

10 

17 

Stud  Crew  Xs 

9.55 

7.71 

9.26  7.88 

8.80 

Op  Crew  Xs 

10.00 

7.14 

7.33  8.33 

8.78 

7.29 

9.41 

X (Student)  = 

= 8.64 

X (Operational)  = 

8.32 

t = 

= 0.55 

p < .599  df  = 

10 

18 

Stud  Crew  Xs 

17.78 

24.14 

20.00  26.96 

26.25 

Op  Crew  Xs 

28.00 

34.29 

33.33  33.33 

30.00 

15.71 

20.00 

X (Student)  = 

= 23.03 

X (Operational)  = 

27.81 

t = 

= -1.33 

p < .21 1 df  = 

10 

19 

Stud  Crew  Xs 

8.89 

4.83 

0.00  5.22 

7.50 

Op  Crew  Xs 

6.00 

4.24 

4.29  2.00 

2.50 

4.29 

3.33 

X (Student)  = 

= 5.29 

X (Operational)  = 

3.81 

t» 

= 1.06 

p < .314  df  = 

10 

20 

Stud  Crew  Xs 

2.33 

1.60 

0.00  0.66 

Op  Crew  Xs 

1.00 

0.66 

1.60  2.00 

4.33 

3.00 

4.00 

X (Student)  =1.15  X (Operational)  = 2 .37 

t = — 1 .48  p < .170  df  = 9 


Appendix  D ( Continued ) 


Variable 

Itt 

Craw 

2nd 

Craw 

3rd 

Craw 

4th  5th 

Craw  Craw 

•th 

Craw 

7th 

Craw 

21 

Stud  Crew  Xs  6.00 

0.00 

6.00 

6.00 

Op  Crew  Xs  1.33 

0.66 

4.80 

1.33  3.33 

0.00 

0.00 

X (Student)  =4.50 

X (Operational)  = 1 .64 

t = 2.01 

p < .192  df  = 9 

22 

Stud  Crew  Xs  4.00 

4.40 

6.00 

5.33 

Op  Crew  Xs  0.00 

0.00 

1.33 

0.00  0.00 

0.00 

4.00 

X (Student)  =4.93 

X (Operational)  = .76 

*t  = 4.97 

p<.001  df  = 9 

23 

Stud  Crew  Xs 

3.40 

7.27  9.00 

Op  Crew  Xs  0.00 

2.00 

3.60 

1.60  2.30 

1.71 

0.00 

X (Student)  = 6.56 

X (Operational)  = 1 .60 

•t  = 3.96 

p <.004  df  = 8 

24 

Stud  Crew  Xs  5.00 

5.00 

0.00 

0.00  1.66 

Op  Crew  Xs  0.00 

5.00 

0.00 

2.73  0.00 

3.33 

10.00 

X (Student)  = 2.33 

X (Operational)  = 3.01 

t = -0.36 

p < .725  df  = 10 

25 

Stud  Crew  Xs  8.75 

9.12 

9.26 

8.33  9.30 

Op  Crew  Xs  7.00 

8.48 

10.00 

8.82  9.41 

8.33 

8.57 

X (Student)  = 8.95 

X (Operational)  = 8.65 

t = 0.65 

P< 

536  df  = 10 

26 

Stud  Crew  Xs  30.00 

30.00 

27.74 

27.50  25.81 

Op  Crew  Xs  28.50 

25.45 

28.89 

25.59  26.47 

26.09 

30.00 

X (Student)  = 28.21 

X (Operational)  = 27.28 

t = 0.88 

p < .596  df  = 10 

27 

Stud  Crew  Xs  37.58 

31.88 

32.45 

29.17  31.63 

Op  Crew  Xs  28.00 

30.00 

35.56 

31.76  33.73 

32.00 

38.57 

X (Student)  = 32.54 

X (Operational)  = 32.80 

t = -0.10 

p < .901  df  = 10 

28 

Stud  Crew  Xs  8.33 

6.00 

5.00 

• 4.33  6.25 

Op  Crew  Xs  0.00 

0.00 

0.00 

6.6  7 7.50 

6.67 

4.00 

X (Student)  = 5.98 

X(Operational)  = 3.54 

t=  1.45 

p < .175  df  = 10 

Not*.  All  t-statistic  numerators  were  computed  X (Student)  - X (Operational). 

•p  < .05. 

Stud  = Student  crews. 

Op  = Operational  crews. 
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APPENDIX  E.  STUDENT  AND  OPERATIONAL  CREW  PERFORMANCE  BY  VARIABLE  ACROSS 
MODES  COMPARED  BY  T-STATISTICS  USING  EQUAL  INTERVAL  SCORING 


VarlaMa 

1ft 

Craw 

2nd 

Craw 

3rd  4th  5th 

Craw  Craw  Craw 

•th 

Craw 

7»h 

Craw 

i 

Stud  Crew  Xs  1.97 

3.21 

2.71  2.91  3.08 

Op  Crew  Xs  2.53 

2.38 

2.04  3.27  1.80 

1.91 

1.96 

X (Student)  = 2.78 

X (Operational)  = 2.28 

t = 1.72 

p < .1 13  df  = 10 

2 

Stud  Crew  Xs  2.21 

2.75 

2.93  2.52  2.86 

Op  Crew  Xs  1.81 

2.71 

2.58  2.03  2.29 

2.41 

2.92 

X (Student)  = 2.65 

X (Operational)  = 2.39 

t = 1.28 

p < .228  df  = 10 

3 

Stud  Crew  Xs  2.52 

2.78 

2.94  2.51  2.91 

Op  Crew  Xs  3.00 

2.79 

2.88  2.89  2.83 

2.81 

3.00 

X (Student)  = 2.73 

X (Operational)  = 2.91 

t = —1.78 

p < .103  df  = 10 

4 

Stud  Crew  Xs  0.00 

0.81 

0.25  0.00  0.00 

Op  Crew  Xs  1 .00 

0.00 

0.17  0.00  1.25 

0.73 

0.00 

X (Student)  = 0.21 

X (Operational)  = 0.45 

t = -.866 

p < .589  df  = 10 

5 

Stud  Crew  Xs  2.57 

2.42 

2.62  1.38  2.14 

Op  Crew  Xs  2.77 

3.52 

3.10  3.44  2.21 

2.00 

3.28 

X (Student)  = 2.33 

X (Operational)  = 2.90 

•t  = -2.04 

p < .049  df  = 10 

6 

Stud  Crew  Xs  1 .0 

0.81 

0.83  0.66  0.76 

Op  Crew  Xs  0.85 

0.92 

1.0  0.96  0.68 

0.73 

0.88 

X (Student)  = 0.81 

X (Operational)  = 0.86 

t = -0.68 

p < .517  df  = 10 

7 

Stud  Crew  Xs  2.25 

2.20 

2.11  134  2.05 

Op  Crew  Xs  2.23 

2.72 

2.25  2.70  1.61 

1.65 

2.28 

X (Student)  = 1 .99 

X (Operational)  = 2.21 

t = 0.86 

p < .586  df  = 10 

8 

Stud  Crew  Xs  1.32 

1.47 

1.49  1.25  1.62 

Op  Crew  Xs  1 .69 

1.84 

2.00  1.85  1.23 

1.87 

1.72 

X (Student)  = 1 .43 

X (Operational)  = 1 .74 

•t  = -2.50 

p < .030  df  = 10 

9 

Stud  Crew  Xs  2.36 

1.71 

2.36  1.75  2.30 

Op  Crew  Xs  2.31 

2.40 

2.60  2.37  2.06 

1.94 

2.64 

X (Student)  = 2.10 

X (Operational)  = 2.33 

t = 1.38 

p < .196  df  = 10 

10 

Stud  Crew  Xs  0.89 

0.71 

0.81  0.63  0.81 

Op  Crew  Xs  0.85 

0.84 

0.95  0.96  0.68 

0.73 

0.88 

X (Student)  = 0.77 

X (Operational)  = 0.84 

t = —1.18 

p < .265  df  = 10 

I 
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Appendix  E ( Continued ) 


Variable 

1ft 

Craw 

2nd 

Craw 

3rd  4th 

Craw  Craw 

9th 

Craw 

•th 

Craw 

7th 

Craw 

11 

Stud  CrewXs 

1.96 

1.50 

1.64  1.31 

1.62 

Op  Crew  Xs 

1.69 

1.84 

2.00  1.93 

1.42 

IAS 

1.83 

X (Student)  = 

1.16 

X (Operational)  = 

= 1.74 

t = 

-0.97 

p < .642  df  = 

= 10 

12 

Stud  Crew_Xs 

0.86 

0.66 

0.62  0.28 

0.73 

Op  Crew  Xs 

0.85 

0.88 

0.45  0.75 

0.84 

0.68 

0.68 

X (Student)  = 

0.63 

X (Operational)  = 

= 0.73 

t = 

-0.98 

p < .648  df  = 

= 10 

13 

Stud  Crew  Xs 

2.44 

1.83 

0.79  1.35 

1.81 

Op  Crew  Xs 

1.60 

1.00 

0.83  1.00 

1.50 

0.92 

1.33 

X (Student)  = 

i.64 

X (Operational)  = 

= 1.17 

t = 

1.79 

p < .101  df  = 

= 10 

14 

Stud  Crew  Xs 

0.22 

0.72 

0.46  1 .00 

1.81 

Op  Crew  Xs 

1.20 

0.43 

0.00  0.10 

0.75 

0.71 

0.50 

X (Student)  = 

0.84 

X (Operational)  = 

= 0.53 

t = 

1.07 

p < .301  df  = 

= 10 

15 

Stud  Crew  Xs 

0.89 

0.86 

0.54  0.70 

0.81 

Op  Crew  Xs 

1.00 

1.00 

0.83  0.80 

1.00 

0.57 

0.67 

X (Student)  = 

0.76 

X (Operational)  = 

= 0.84 

t = 

-0.83 

p<  .569  df  = 

= 10 

16 

Stud  Crew  Xs 

2.33 

1.79 

1.54  1.96 

1.94 

Op  Crew  Xs 

1.60 

2.14 

1 .83  1 .56 

2.47 

1.79 

1.67 

X (Student)  = 

1.91 

X (Operational)  = 

= 1.87 

t = 

0.29 

p < .77  df  = 

= 10 

17 

Stud  Crew  Xs 

0.95 

0.77 

0.93  0.79 

0.88 

Op  Crew  Xs 

1.00 

0.71 

0.73  0.83 

0.88 

0.72 

0.94 

X (Student)  = 

0.86 

X (Operational)  = 

= 0.83 

t = 

0.56 

p < .593  df  = 

= 10 

18 

Stud  Crew  Xs 

0.89 

1.21 

1.00  135 

131 

Op  Crew  Xs 

1.40 

1.71 

1.67  1.67 

1.50 

0.79 

1.00 

X (Student)  = 

1.15 

X (Operational)  = 

= 1.39 

t = 

-1.33 

p < .21 1 df  = 

= 10 

19 

Stud  Crew  Xs 

0.89 

0.48 

0.00  0.52 

0.75 

Op  Crew  Xs 

0.60 

0.43 

0.43  0.20 

0.25 

0.43 

0.33 

X (Student) = 

0.53 

X (Operational)  = 

= 0.38 

t = 

1.05 

p < .38  df  = 

= 10 

20 

Stud  Crew  Xs 

2.33 

1.60 

0.00  0.00 

none 

Op  Crew  Xs 

1.00 

0.60 

1.60  2.00 

4.32 

3.00 

4.00 

X (Student)  = 

0.98 

X (Operational)  = 

; 2.37 

t = 

-1.63 

p < .135  df  = 

9 
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Appendix  E ( Continued ) 


Variable 

1st 

Craw 

2nd 

Craw 

3rd  4th  5th 

Craw  Craw  Craw 

6th 

Craw 

7th 

Craw 

21 

Stud  Crew  Xs  2.00 

Op  Crew  Xs  0.66 

0.00 

0.33 

2.00  2.00  none 

2.40  0.66  1.66 

0.00 

0.00 

X (Student)  = 1 .50 
t = 1.17 

X (Operational)  = 0.82 
p < .272  df  = 9 

22 

Stud  Crew  Xs  2.00 

Op  Crew  Xs  0 00 

2.20 

0.0 

3.00  2.66  none 

0.0  0.66  0.00 

0.00 

2.00 

X (Student)  = 2.47 
*t  = 4.97 

X (Operational)  = 0.38 
p<.001  df  = 9 

23 

Stud  Crew  Xs  none 

Op  Crew  Xs  0.00 

none 

1.50 

1.90  4.27  5.00 

3.00  1.60  1.80 

1.57 

0.00 

X (Student)  = 3.72 
*t  = 2.81 

X (Operational)  = 1.35 
p < .022  df  = 8 

24 

Stud  Crew  Xs  0.50 

Op  Crew  Xs  0 00 

0.50 

0.50 

0.00  0.00  0.17 

0.00  0.27  0.00 

0.33 

1.00 

X (Student)  = 0.23 
t = -036 

X (Operational)  = 030 
p < .725  df  = 10 

25 

Stud  Crew  Xs  0.88 

Op  Crew  Xs  0.70 

0.91 

0.85 

0.93  0.83  0.93 

1.00  0.88  0.94 

0.83 

0.86 

X (Student)  = 0.90 
t = 0.66 

X (Operational)  = 0.87 
p < .532  df  = 10 

26 

Stud  Crew  Xs  1 .00 

Op  Crew  Xs  0.95 

1.00 

0.85 

0.92  0.92  0.86 

0.96  0.85  0.88 

0.87 

1.00 

X (Student)  = 0.94 
t = 0.89 

X (Operational)  = 0.91 
p < .602  df  = 10 

27 

Stud  Crew  Xs  0.94 

Op  Crew  Xs  0.70 

0.80 

0.75 

0.81  0.73  0.79 

0.89  0.79  0.84 

0.80 

0.96 

X (Student)  = 0.81 
t = -0.10 

X (Operational)  = 0.82 
p < .901  df  = 10 

28 

Stud  Crew  Xs  1 .67 

Op  Crew  Xs  0.00 

1.20 

0.00 

1.00  0.87  1.25 

0.00  1 33  1.50 

1.33 

0.80 

X (Student)  = 1.20 
t = 1.45 

X (Operational)  = 0.7 1 
p < -175  df=  10 

Not*.  All  t -statistic  numerators  were  computed  X (Student)  - X (Operational). 
*p  < .05. 
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